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The evaluation design for the Austin Independent 
School District 1982 summer school program for elementary retainees 
is described. The evaluation is meant to focus on short-term skill 
mastery in reading and mathematics and the long-term achievement 
growth of the participants. Separate reading objectives and materials 
in Spanish and English language arts for Limited English Proficiency 
students are planned. Among the relevant evaluation questions and 
objectives to determine program effectiveness are whether the 
structure of summer school will be appropriate for future 
application, and whether alterations will be necessary. Information 
sources for data to be used in projected decisions will include 
student mastery files; project files and records; teacher, parent, 
and director surveys; Iowa Tests of Basic Skills files; observations; 
and mastery tests. Evaluation activity time resource allocations and 
dissemination plans are provided. The AISD Office of Research and 
Evaluation will report findings in the fall of 1982 and the summer of 
1983. (CM) 
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Evaluation Design ABSTRACT 

title: EVALUATION DESIGN: Summer School for Retainees, 1982 
Contact Persons: Nancy Baenen, John MacDonald 
No. Pages : 16 
Summary: 

The evaluation design describes the evaluation plan for this project. It 
includes a brief project and evaluation summary, major decision and evalua- 
tion questions to be addressed, dissemination plans, information sources 
to be used, data to be collected in the schools, and evaluation resources. 

The summer school for retainees will be held June 7 through July 9, 1982. 
It will include reading, mathematics, and recreational activities in a 
four-hour program each day. Mastery learning materials will be used in 
both reading and mathematics . Limited English Proficiency students in 
Categories A and B will have separate materials for Spanish and English 
language arts. The evaluation of the summer school program will focus on: 

• Short-term skill mastery in reading and math of participants; 

• Long-term achievement growth in reading and math of participat- 
ing retainees versus non-participating retainees; 

• Characteristics of the program which may relate to program 
effect* zeness. 

Separate reading objectives have been set for LEP students and some separate 
assessment will be necessary. 

The Office of Research and Evaluation will report findings in fall of 1982 
and again in summer 1983. 
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EVALUATION DESIGN REVIEW FORM 



The iudivi duals listed below were provided an opportunity to review this 
design and provide input prior to its publication. 

Lawrence Buford 

Acting Associate Superintendent for Instruction 
Ruth MacAllister 

Assistant Superintendent for Elementary Education 
Timy Baranoff 

Director, Elementary School Curriculum 
Hermelinda Rodriguez 

Director, Elementary School Management 

Paola Zinnecker 
Instructional Coordinator 

Anita Uphaus 

Instructional Coordinator 
Lucy Sahraie 

Instructional Coordinator 

Joan Burnham 
Educational planner 

Lee Laws 

Director, Applications and Compliance 
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EVALUATION SUMMARY 



The evaluation of the Austin ISD summer school for retained elementary 
students will focus on the following areas related to program effective- 
ness: 

• Short-term skill mastery in reading and math of participants; 

• Long-term achievement growth in reading and math of participat- 
ing retainees compared to non-participating retainees;, 

• Characteristics of the program which may relate to program effec- 
tiveness. ; 

Short-term growth of both LEP and non-LEP retainees during summer school will 
be measured through criterion-referenced mastery tests for each skill unit. 
Long-term achievement growth of non-LEP students will be measured using the 
reading and mathematics sections of the Iowa Tests of Basic Skills (ITBS) . 
Long-term growth of LEP students 1 reading skills will be assessed by means of 
a Spanish-language achievement test. For non-LEP students, test results 
from the general administration of the ITBS in April 1982 will be used as the 
pretest; those from April of 1983 will be considered the posttest. Perform- 
ance of retainees who did and did not participate in the summer school will 
be compared in April 1983 after the next general test administration. 

Process information will be collected through structured observations of a 
random sample of students. Students' mastery tests and teacher records will 
also be reviewed for relevant information. Central computer files will supply 
most of the descriptive information necessary on student and staff character- 
istics. 
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PROGRAM SUMMARY 



The AISD 1982 summer school program will be held from June 7-July 9. It will 
be open to all AISD students in grades 1-6 that have been retained at any 
point in their school careers. The basic summer school program will include: 

Reading /Language Arts 1% hours 

Recreational Activity 1 hour 

Math and Applied Skills lh hours 

The 1982 summer school includes a number of new features designed to ^promote 
program effectiveness: 

■ Staff will receive five days of staff development. Three will 
orient them to the curriculum and overall program. The addi- 
tional two days will allow time for them to review descriptive 
arid skill information on each student forwarded from the previous 
teacher, call one-quarter of the students' previous -teachers to dis- 
cuss their needs jnore fully, and visit the homes of one-eighth of 
the students to gain a different perspective and elicit parent sup- 
port and involvement. It is hoped that this provision of informa- 
tion will decrease assessment time during summer school, build 
rapport more quickly, and increase parent involvement. The evalu- 
ation will collect information on the successf ulness of this 
activity and its relationship to achievement. 

• Mastery learning systems will be piloted in both reading arid math. 
These structured materials provide criterion-referenced tests to 
measure mastery of skills. The Chicago Mastery Learning System 
will be used for reading and Region Kill's Math for" Everyone will 
be used for mathematics . These materials may be used during the 
regular school year if the pilot shows them to be effective. 

• Students in Limited English Proficiency reading .classes will receive 
instruction in Spanish reading using the Caracolitos and Una Cosa 
reading series, and instruction in English as a Second Language 
using the Stepping Into English series. These students will receive 
math instruction in Spanish using the Math for Everyone materials. 
Mastery tests from Math for Everyone will be used to test these 
students in math, and criterion-referenced tests from Stepping 

Into English and from the Spanish reading series will be used to 
.assess these students' progress in reading.. 
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• Award systems for performance and attendance aife planned to serve 
as motivators. Different awards will probably Sbe used across 
classes to allow tests of effectiveness. 1 

• Followup activities have been planned in both reading and math to 
helf) fill the gap between summer school and the regular school 
year. The amount of structure and the method of presentation will be 
varied to allow comparisons of effectiveness. 
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Ill A 

DECISION QUESTIONS ADDRESSED 



System-Level Decision Questions \ 

1. Were the students served and staffing of summer school appropriate 
for future summer schools? Are alterations necessary? 

2., Was the structure of summer school appropriate for future summer 
schools? Are alterations necessary? 

3. Should additional information be provided to teachers about the 
students before the start of future summer school programs? 

4. Should retainees bef encouraged to attend summer school?- 
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DECISION QUESTIONS OVERVIEW 



DECISION QUESTION 


DECISION 
DATE 


DATE 
NEEDED 


RELEVANT EVALUATION 
QUESTIONS & OBJECTIVES 


INFORMATION SOURCES 


Dl, Were the students served and staffing 
of summer school appropriate for 
future summer schools? Are altera- 
tions ijecessary? 

t 


Fall 1982 


August 1982 

> 


0 

1. What were the chsracterlstlcs 
of students served by summer 
school including: 

- Age 

- Sex 

- Grade level 

- Ethnicity 

- Ellglblli** for Title I, 
Title I u rant, LEP, and 
SC£ progr^ns in 1981-82 

- Service by Title^I, Title I 
Migrant , blllngtfal , and 5CE 
programs in .198 1-^2 

- Number of years enrolled in 
A1SD 

- Attendance rate for 1981-82 

- Attendance In summer school 

- Eligibility for Special Educa- 
tion services 

- Year of retention: tht year 
^ or earlier 

- Historical achievement data: 
mean scores in grade equiv- 
alents on 1TBS in reading 
and math, spring 1982 


Student Master File, 

i 

ORE Project Files 
n. 

Teacher Records 

Teacher Records 
Teacher Records 
Student -Master File 

ITBS Files 




Fall 1982 


August L982 


2. What were the characteristics 
of participating staff by: 

- Sex 

- Ethnicity 

- Years of experience in 
education 

- Educational background 
- - Certification 


Employee Master File 

*7 
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DECISION QUESTIONS OVERVIEW 



DECISION QUESTION 


DECISION 


DATE 


RELEVANT EVALUATION 


INFORMATION SOURCES 

1 II ll VI llllltl 1 V/ 1 » V/ \-# ■ 1 V> *— 




DATE 


NEEDED 


QUESTIONS & OBJECTIVES 




D2. Was the structure of summer school 


Fall 1982 


August 1982 


I. What training did staff 


Teacher Survey 


appropriate for future summer 






receive? Did staff feel the 


fW rprhnr StlfVPV 
uiicLLUi uui vcjr 


schools? Are alterations necessary? 






training was effective? 










2. What did the math curriculum 


ProJ ect Records 








include?. 




• 






(Including materials and equip- 










ment needed) 










3. What did the reading curriculum 


Project Records 








include? (Including materials 










and equipment) 


c 








4. Who planned the program and 


Project Records 








what aspects did they organize? 










5. How were student learning needs 


Project Records 








assessed and monitored? 


Observations 








6. What was the planned and actual 


Project Records 








pupil/teacher ratio? 


Observations 








7. How much of the math and read- 


Observations 


i 






ing allotted time was spent 










on task? 










8. Were award systems implemented 


Project Records 








as planned? How many students 


Observations 








earned the awards? 










9. How much material were the 


Teacher Records 








teachers able to cover In math 


Mastery Tests 








and reading? How long did 










reading units take to teach? 
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Ill B 

DECISION QUESTIONS OVERVIEW 



00 



DECISION QUESTION 


DECISION 
DATE 


DATE 
NEEDED 


RELEVANT EVALUATION 
QUESTIONS & OBJECTIVES 


INFORMATION SOURCES 








1Q„- How. wuch .Lime d i<L st ud e n t s 
interact with the teacher? 
How much time did students 
work on their own? 

11. How did parents like summer 
school? 


Observations 
Parent Survey 








12. What effect did the home 
visit have on parents 1 
activities with their children? 

13. Did parents receive information 
about activities to do with 
their children for the rest 

of the summer after summer 
school was completed? How 
much did they complete? 


Parent Survey 
Teacher Survey 

Parent Survey 
Project Records 


D3. Should additional information be 
provided to teachers about the 
students before the start of future 
summer school programs? 


Sprint? 1983 


Kali 1982 


1. Did teachers receive informa- 
tion on retainees from the 
previous teacher? When? 

2. Were summer school teachers 
able to reach regular school 
teachers of assigned retainees? 

3. Were teachers able to visit 

f he homes of assigned retain- 
res? 

A. How valuable were the home 
visits perceived to be by 
summer school teachers? 


Teacher Records 

Teacher Records • 
Teacher Survey 

Teacher Records 
Teacher Survey 
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DECISION QUESTIONS OVERVIEW 



DECISION QUESTION 


DECISION 
DATE 


DATE 
NEEDED 


RELEVANT EVALUATION 
QUESTIONS & OBJECTIVES 


INFORMATION SOURCES 








5. How much tine did teachers 
spend in assessing students' 
skills? Did teachers spend 
more time on this activity 
during the first week of 
summer school? 


Observations 


D4. Should rtita inees be encouraged to 
attend summer school? 


Spring 1983 


Spring 1983 


]. Did students meet short-term 
objectives? How many skills 
were the students able to 
master (at an 80% level) In 
reading and math by the end 
of summer school? 

Short-term objectives: 
Reading: By the end of the 
five-week summer school, reading 
skills specified for each grade 
level will be mastered by 90% 
of the retalnees participating. 


• 


/ 






LKP Reading: Spanish Reading: 
LEI* retalnees participating In 
summer school will show 80% 
accuracy on workbook assessments 
on the average. 
English as a Second Language: 
LKP retalnees participating 
In summer school will complete 
at least two levels In the 
Stepping Into English series. 

Math: By the end of the five 
week summer school, participat- 
ing retalnees will, on the aver- 
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Ill B 

DECISION QUESTIONS OVERVIEW 



DECISION QUESTION 


DECISION 
DATE 


DATE 
NEEDED 


RELEVANT EVALUATION 
QUESTIONS & OBJECTIVES 


INFORMATION SOURCES 








age, master the number of skills 
specified for their Instruction- 
al level at an 80% level. 

2. Did students meet long-term 
objectives? 

Long-term objectives; 
Reading; As of April 1983, 
retainees participating In the 
1982 summer school will allow 
higher achievement in reading 
areas emphasized than will 
retainees who 'did not partici- 
pate based on the Jowa Test of 
Basic Skills (ITBS). 

Math: As of April 1983, retain- 
ees participating in the 1982 
summer school will show higher 
achievement in math areas emp- 
hasized than will retainees 
who did not participate based 
on the lowa Test of Basic Skills 
(ITBS) . 

3. What were the average grade 
equivalent scores of retainees 
in the summer school in April 
1982 and April 1983? By 
skill areas emphasized and not 
emphasized in summer school? 


ITBS Files 

Curriculum/ITBS Skills Match 
ITBS Files 

Curriculum ITBS Skills Match 
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DECISION QUESTIONS OVERVIEW 



DECISION QUESTION 


DECISION 
DATE 


DATE 
NEEDED 


RELEVANT EVALUATION 
QUESTIONS & OBJECTIVES 


1 INFORMATION SOURCES 








A. How did the scores of retain- 
ees who attended summer school 
compare to those of retalnees 
who did not attend summer 
school aa of April 1983? By 
skill areas emphasized and not 
emphasized In summer school? 

5. Can any variables be Identified 
that relate to student achieve- 
ment? 

1 


ITBS Files 

Curriculum/ ITBS Skills Match 

ITBS Files 
Parent Survey 
Observations 
Project Records 
Teacher Records 
Teacher Survey 
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INFORMATION SOURCES 



INFORMATION 
SOURCE 


POPULATION 


EVAL. QUES. 
REFERENCED 


DATE 
COLLECTED 


ANALYSIS 
TECHNIQUES 


REMARKS 


Student Master File 


All AISD students 


Dl-l 


July 1982 


Descriptive statistics on 

ClIQl QClcl lBllCo Ul bLUUcilLU 

served* 




ORE Project Files 


All students eligible for 
and served by categorical 
programs at the elementary 
level 


Dl-1 


July 1982 


Descriptive statistics on 
program participation 
u u ring i7o i oa 


Title I, Title I 
Migrant, r ,EP, and SCfi 
Files 


Teacher Records 


Classroom teachers* records 


Dl-l, u^-y , D J-i , 
D3-2, D3-3, D4-5 


June— July l?oz 


Lies crip tive bcqi.xbi.xc8 
summarizing data 


Vporq pnrnllpri attend— ' 

ance rates, material 
covered, home visits 
attempted, accomplished, 
phone calls to former 
teachers attempted, 
accomplished 


ITBS Files 


All elementary students 


Dl-l , D4-2, D4-3, 
D4-4, 1)4-5 


July 1982 
April 1983 


Descriptive statistics on 
test scores (mean, standard 
deviation, range, etc.), 
regression analyses 




Employee Master File 


All teachers in AISD 


Dl-2 


July 1982 


Descriptive statistics 
on staff characteristics 




Observations 


Teachers and a sample of 
summer school students 


D2-5, D2-6, D2-7, 
1)2-10, D3-5, D4-5 


June-July 1982 


Amount and percent of time 
spent in various activities 




Project Records 


Overall records for project 
general information 


D2-2, D2-3, D2-4, 
D2-5, D2-6, D2-8, 
D4-5 

- 


June-July 1982 

1 - 


Summary of information 
collected — narrative and 
quantitative 


Curriculum, planned PTR, 
grouping, award systems, 
follow-up, key variables 
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INFORMATION SOURCES 





INFORMATION 
SOURCE 


. POPULATION 


EVAL. QUES. 
REFERENCED 


DATE 
COLLECTED 


ANALYSIS 
TECHNIQUES 


REMARKS 




Mastery Tests 


All summer school students 


D2-9, D4-1 


June-July 1982 


Tallies of skills mastered, 
percent mastery, other 
descriptive statistics 


Information needed varies 
for reading snd math and 
for LEP reading 




Teachur Survey 


All summer schot/1 teachers 


D2-!, D2-L2, 
D3-2, D3-3, D3-4, 
D4-5 


July 1982 


Descriptive statistics: 
number and percent by 
response cstegory, means 






Director Surve/ 


All summer school directors 


D2-1 


July L982 


Descriptive statistics 






Parent Survey 


The parents of a random 
selection of summer school 
students 


D2-LL, I>2-12, 
D2-13, D4-5 


August 1982 


Descriptive statistics 






CurricuVum/lTBS 
Skills Match 


Instructional Coordinators 


D4-2, 1)4-3, D4-4 


May-June 1982 


Summary of results of 
skills analysis of ITBS 
snd summer school curricu- 
lum 
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DATA. TO BE COLLECTED IN THe)sCHOOLS 

June-July 1982 A. Students 

1. Observations of a random sample of 
summer school participants. 

2. Mastery tests given by the classroom 
teacher for each skill unit (part of 
regular curriculum) . 

B. Teachers 

1. Teachers 1 records will be reviewed for 
information on skills mastered by stu- 
dents. Cards provided by regular teacher 
will provide ID number, attendance and 
enrollment in AISD data, 

2. Teacher survey — Teachers will be asked 
about the effectiveness of training ( 
provided, the curriculum materials, and 
summer school overall. 

Cm Administrators 

1. Director survey — Administrators will be 

asked a. few general and specific questions 
about summer school, award systems, etc, 

D, Parents 

1. Parent Survey — A sample of parents of 

summer school participants will be asked 
* about followup activities completed and 
summer school in general. 
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VIII 

EVALUATION TIME RESOURCES ALLOCATION 



ACTIVITY 


DIRECTOR 


EVALUATOR 


EVALUATOR 
INTERN 


PROGRAMMER 


OBSERVERS/ 
TESTERS 


SECRETARY 


A. 


Design 


.5 


7 


1 






2 


B. 


Information Sources 




U • D 












Student Master File 




U • J 


1.5 


1.5 








ORE Project Files 




U • j 


1.5 


2.0 








Teacher Records 




1 • U 


4 




9 






ITBS Files 




1 .U 


15 


3.0 




.5 




Employee Master File 




U . 5 


2 


0.5 








Observations 


.5 


1.0 


14 


0.5 


52 


.5 




Project Records 




1.0 


5 




8 


%5 




Mastery Tests 




0.5 


4 


- 


6 


.5 




Staff Survey . 


.5 


0.5 


3 


0.5 




.5 




Parent Survey 




0.5 


2 


0.5 




.5 




Gurriculum 


.5 


1.5 








1 




ITBS Skills Match 














C. 


November Technical Report 


1 


2.0 


16 






8 


D. 


November Summary 


.5 


1.0 


3 






1 


E. 


April Technical Report 


,5 


3.0 




2.0 




2 


F. 


Administrative and Other Indirect 
















Costs 


2 


6.0 


8 


1.0 




1 


- TOTAL DAYS 

27 


6 


28.0 


80 


ia.5 


75 

i 


18 
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DISSEMINATION 



INFORMATION 


DISSEMINATION 


DATE 


PERSONS 




FORMAT 




RECEIVING 


Interim Report on all findings 


Technical Report 


November 


TEA, project staff 


except April 1983 ITBS test 


with Executive 


1982 


and key adminis- 


results 


Summary 




LlaLUlO ^ piUl CO 








>J J-U lid JL JL JL Lf ±. a i- j 


Summary of i esults 


Executive 


November 


Selected AISD 




Summary, bro- 


1982 


adminis trator s , 




chure, or news- 




qtimmp t cjpfinnl 
o uumc i- o v^ittju jl 




letter 






Final report of April 1983 find- 


Retention Tech- 


July 1983 


TEA, key AISD 


ings on ITBS results 


nical Report 




administrators , 








professional 








library 




Retention sec- 


July 1983 


AISD administra- 




tion of 1982-83 




tors, professional 




Findings Volume 




and school 








libraries 
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